Engineering posts about High Availability
Curated summaries and key learnings for engineers working with High Availability.
How AI-Powered Attacks Led Salesforce to Reinvent Hyperscale DDoS Defense
The article outlines the development of DREAM, an AI-powered DDoS mitigation platform designed by Salesforce to enhance its defense against hyperscale DDoS attacks. It highlights the architectural...
MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet
MetaRoCE is a newly developed RDMA transport protocol specifically designed for AI workloads over commodity Ethernet, aiming to enhance the performance and reliability of data transfer between GPUs...
Databricks Network Configuration delivery to Tens of Millions of Serverless VMs
The article discusses the re-architecture of Databricks' network configuration delivery system for its serverless platform, which launches tens of millions of virtual machines (VMs) daily. The...
How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…
In this article, Netflix engineers Nilesh Mishra and Ajit Koti detail the architecture and design choices behind their Real-Time Distributed Graph (RDG), particularly focusing on the querying layer...
How Salesforce Eliminated Single-Region Risk and Reduced Downtime Blast Radius at 4B Metrics/Min
The article details Salesforce's engineering advancements in their observability platform, Argus, which processes approximately 4 billion metrics per minute. It highlights the transition from a...
Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned
This article delves into the complexities of constructing a real-time service topology at Netflix scale, focusing on the architectural decisions that facilitate efficient data processing and system...
Introducing Meerkat: an experiment in global consensus
The article introduces Meerkat, a new distributed consensus service developed by Cloudflare, designed to manage control-plane state across its global data centers. It highlights the limitations of...
From Silos to Service Topology: Why Netflix Built a Real-Time Service Map
The article outlines Netflix's development of a real-time service topology map to improve observability and troubleshooting in its microservices architecture. It highlights the challenges faced by...
How Netflix Simplified Batch Compute with Kueue
The article outlines Netflix's transition from its custom Compute Managed Batch (CMB) solution to Kueue, a cloud-native job queueing system designed for batch workloads. This migration was motivated...
Scaling Zero Copy from 1 Trillion to 120 Trillion Rows with File Federation
The article details the evolution of Zero Copy architecture at Salesforce, transitioning from a Query Federation model to a File Federation architecture to support AI workloads at petabyte scale...
Sitar-agent: Building a reliable dynamic configuration sidecar at scale
The article discusses the development of Sitar-agent, a Kubernetes sidecar designed to ensure reliable dynamic configuration delivery at scale for Airbnb's services. It outlines the configuration...
Lights Out, Systems On: Validating Instant Power Loss Readiness
The article introduces the Instantaneous PowerLoss Storm, a testing paradigm developed by Meta to prepare data centers for zero-notice power loss scenarios. It outlines the strategies implemented to...
Improve your application resilience with Amazon Cognito multi-Region replication
The article discusses the introduction of multi-Region replication for Amazon Cognito, aimed at enhancing application resilience by maintaining consistent user authentication across AWS Regions. It...
How the lakebase architecture stays resilient to cloud failures
The article discusses the challenges faced by cloud infrastructure due to increased demand for control-plane operations and the need for high availability in database management. It outlines how the...
Rethinking Distributed Systems for Serverless Performance and Reliability
The article explores the evolution of serverless compute for Apache Spark, addressing long-standing architectural challenges that have hindered performance and reliability. It emphasizes the need for...
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
The article outlines Slack's transition from a legacy SSH-based architecture to a modern REST-based job submission system for its data pipelines. Initially, the reliance on SSH created significant...
Building a fault-tolerant metrics storage system at Airbnb
The article details Airbnb's development of a high-throughput metrics storage system capable of ingesting 50 million samples per second and managing 2.5 petabytes of data. It outlines the challenges...
Rearchitecting the Workflows control plane for the agentic era
The article discusses the rearchitecting of the Workflows control plane to accommodate a shift towards agent-triggered workflows, necessitated by the increasing demand for durable execution engines...
Building a Distributed Persistent Queue That Scaled AI Workloads 5x Under LLM Rate Limits
The article discusses the engineering of a distributed persistent queue that orchestrates AI workloads and human workflows within strict infrastructure limits. It highlights the challenges of scaling...